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Abstract 


This study reports on the development of a cooperative learning attitude scale for the EFL students based on 
expert reviews and examines its construct validity. Three rounds of expert reviews were conducted. In each 
round, five content experts reviewed the items and rated their relevance to the content domain and their compre- 
hensibility. The expert panel also assigned the items to the suggested sub-scales. After item elimination and 
major revisions, 45 items representing five factors with perfect reviewer consensus were retained. Although the 
scale was developed via expert review, the authors checked the construct validity of the instrument through CFA 
by collecting data from 136 pre-service EFL teachers. The results suggested elimination of six more items owing 
to their insignificant t values or low factor loadings, which were eliminated after two reviewers recommended to. 
The resultant fit statistics supported a good-fitting model with t statistics and item-total correlation significant 
across all variables. The coefficient alpha confirmed internal consistency of the scale and that of sub-scales. 
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1. Introduction 


‘Language acquisition’ is vastly being highlighted as a significant alternative to the ‘lan- 
guage learning’ in foreign/second language education. There is a distinction between the two 
terms. That is, learning a language is seen as a deliberate and intentional process, while its ac- 
quisition is considered to happen in the opposite way. It is argued that the language acquisition 
takes place unintentionally by means of frequent social interactions (Mitchell et al., 2013). Ifa 
language is acquired it comes automatically when needed, but if it is learned consciously then 
there will not be this automaticity and the learners ought to verify what they want to utter in 
their mind first to check for accuracy (Harmer, 2007). Children, for example, exhibit an 
astounding level of aptitude in acquiring more languages in addition to their mother tongue 
without actually being taught about the forms or vocabulary but acquire them through commu- 
nication in a particular social environment. However, Harmer argues that the language acquisi- 
tion process weakens by age growth. Nevertheless, getting the right input in stress-free learning 
atmosphere individuals can acquire a language even in their adolescence. This can be made 
possible by the amount of “...exposure to it, motivation to communicate with it and opportuni- 
ties to use it” (Harmer, 2007, p. 47). 


According to Ur (2012), the second languages could be acquired intuitively in the same way 
as the native languages are acquired through repetitive exposure to social interaction and com- 
munication with others. Truly, “[to] learn a second language is to learn a skill” and it requires 
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extensive practice (McLaughlin, 1987; cited in Pienemann, 2003, p. 681). In some cases, if the 
teachers only focus on acquisition and always give it a priority, students will remain uninformed 
about the structures and functionality of the target language, especially in adult language educa- 
tion. Students require explicit teaching of the structures and forms of the language by the teach- 
er. Here it is that they have to learn consciously about the language structures (Mitchell et al., 
2013). However, the focus of the language acquisition theorists is on the language learners to 
acquire their target language through social interactions; even the grammar of that language 
alongside its respective language skills. In fact, “language is social — a social practice, a social 
accomplishment, a social tool” and thereby must be acquired through social interactions (Atkin- 
son, 2002, p. 526). 


From a cognitive theory perspective, language acquisition is seen as an internal and mental 
process. Cognitive theorists believe that human beings come with innate language acquisition 
abilities. To them, language acquisition occurs by means of social interactions, which, in turn, 
will facilitate the internalization and discernment of the linguistic features even in L2. Moreo- 
ver, the cognitive theory is considered as an underlying theory for the other most important 
emerging theories in second language acquisition research area (Atkinson, 2011). Here, soci- 
ocultural theory to language acquisition seems to meet the purpose of the L2 acquisition in col- 
lege education. 


Sociocultural theory focuses on the students’ exposure in in-class social interactions recom- 
mending their active engagement in certain activities. No doubt, it has a profound impact on 
their L2 development thereafter. A distinguishing trait of this theory from the other alternatives 
is that “‘...it does not assume that acquisition is a universal process” (Lantolf, 2011, p 43). It sees 
the L2 acquisition process differently according to the various social and cultural settings. The 
sociocultural theory also encourages direct explanation of various ‘linguistic features’ of the 
target language, so that the students can apply them in their communicative activities. Language 
learners’ mediating characteristics in language acquisition process might be different. For in- 
stance, students may later apply the explicitly learned grammar implicitly for their communica- 
tive needs in their own ways based on their capabilities in mediating different linguistic fea- 
tures. Gillies and Boyle (2008) report that “by encouraging students to share their ideas with 
their peers, clarify their misconceptions, and work together to construct new understandings, the 
teacher...” provides them “mediated-learning” opportunities when they are working in coopera- 
tion with each other (p. 1346). Pawlak (2011) calls this process “instructed foreign language 
acquisition”. He contends that the foreign language acquisition is limited to the classroom 
teaching and the students get little exposure to the language practice in their surroundings out- 
side the classroom context. As noted elsewhere, the foreign language students need plentiful 
exposure to the natural target language practice in cooperation with their peers within the class 
through the guidance of the language instructor. The use of cooperative learning in language 
acquisition is elaborated in more details beneath in the literature review. 


2. Literature 


2.1. Cooperative learning and language acquisition 


Cooperative learning (CL) comes with a variety of methods into effect as an effective ap- 
proach in foreign/second language acquisition seemingly serving the purpose of sociocultural 
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learning theory, where the students both receive input from the instructors as well as interact 
within cooperative groups to enhance one another’s learning. According to Slavin (1991), CL 
“supplements the teacher’s instruction by giving students an opportunity to discuss information 
or practice skills originally presented by the teacher”, while sometimes they are also required to 
locate the sources of information by their own efforts to have contributed to the accomplishment 
of their shared cooperative tasks (p. 73). This is exactly in congruence with the discussion made 
above as regards the implication of sociocultural theory in an L2 acquisition context. 


Keeping the contextual difference of foreign language education in mind from that of native 
languages, the CL methods could provide a stimulatory simulative learning environment where 
the learners get encouraged in actual social interaction to promote their language skills in coop- 
eration with each other by building small communities of learners. So doing, they can benefit 
from the comprehensive inputs they receive through teacher mediated opportunities to com- 
municate with and use the target language during the collaborative activities with their peers. 
Learners’ interaction within a group of L2 learners is, in fact, a sort of simulation in the target 
language community that they will interact someday and it plays as a motive for them to be 
persuaded to have full participation in communicative activities in the class (Norton & McKin- 
ney, 2011). 


Kagan and Kagan (2009) maintain that besides developing social and higher-order thinking 
skills, CL serves the purpose of language acquisition. They further argue that in a well-tailored 
CL environment all the students with no exception get enough exposure to the target language 
practice by using it in self-disclosure, exchanging ideas and negotiating. This way, students in 
all teams in the class will be improving their listening and communicative language skills. As a 
matter of fact, “students learning a new language are more willing to participate and preserve in 
a cooperative versus competitive setting” especially in college education (Kagan & Kagan, 
2009, p. 3.6). Besides, CL plays a crucial role in increasing self-esteem, and motivation, while 
also decreasing student language anxiety. 


In addition, CL puts an incremental positive impact over the academic achievement of the 
students compared to the competitive or individualistic approaches to learning irrespective to 
the grade level or subject area (Johnson, Johnson, and Stanne, 2000; Slavin, 1996). The 
achievement of the students in CL groups is interrelated with the level of motivation students 
get while working on accomplishing a shared goal with their peers in small groups. In CL ac- 
tivities, students are required to be responsible for their learning and that of others in their group 
(Slavin, 1991, 1996). 


As a motivational attribute, Slavin (1996) emphasizes the presence of group rewards or 
stimuli like scores on student individual performance in the quizzes. This will also play an im- 
portant role in student cohesiveness in cooperative group works. Contrarily, Kagan and Kagan 
(2009) are against the group or individual rewards of such kinds and believe that the motivation 
will be temporary in such cases. When the rewards diminish, the motivation will also diminish. 
Instead, they suggest immediate praising and constructive feedback to acknowledge students 
efforts towards their own learning and that of their teammates while the cooperative group task 
is running. Likewise, Baer (2003) maintains that rewards may work in school level, but will not 
have any impact on college students in tertiary level CL activities. 
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2.2.Cooperative group design 


Cooperative group works are different from normal groups where students do not feel con- 
nected to each other and are unmotivated. In cooperative learning, students are required to work 
in small groups to accomplish a common group goal. They have to help each other learn an 
intended content knowledge through discussions ensuring the active contribution and hard work 
of every single member of the group in the process (Johnson & Johnson, 1999). CL tasks must 
be completed by exerting efforts both in an individual and group level (Schul, 2011). 


The cooperative groups are formed on heterogeneity basis, where students with differences 
“in skill level, interest, motivation, experience, and family backgrounds” work together (Apple, 
2006, p. 279). They invest in each other’s learning and feel proud of having a positive role in 
the success of their group members while also believe in their own potentials in carrying out the 
tasks they are assigned to (Johnson & Johnson, 2003). Meanwhile, the students’ equal participa- 
tion in teamwork irrespective of differences in terms of social rank and potential aptitudes is of 
primary importance in CL (Apple, 2006). So doing, they will learn to see things from different 
viewpoints, build s strong rapport with one another, learn to work with others, and start being 
optimistic as regards their learning process (Favor, 2012). 


One more thing that is worth to mention is that the students are better aware of one another’s 
problems than the teacher is with the new learning material they are dealing with and can better 
help each other understand thereafter (Gillies & Boyle, 2008). Likewise, their contribution will 
increase, when they know their standpoints in group discussions will be treasured by their peers. 


The elements of successful cooperative group work are five as suggested by Johnson and 
Johnson (1999, 2002): 


Positive Interdependence: students must believe that their success is interrelated to the suc- 
cess of the entire group. The task they are required to accomplish cooperatively is based on a 
common goal equally and mutually benefits everyone in the group. 


Individual Accountability: to ensure that everyone in the groups has improved during the co- 
operative learning, it necessitates individual assessment that, in turn, will cultivate a sense of 
individual responsibility in students to contribute to a cooperative shared task; especially when 
the result of the assessment is shared with the group or the individual students. 


Face-to-face promotive interaction: is referred to the contribution of the students to the pro- 
motion of one another’s achievement “by helping, assisting, supporting, encouraging, and prais- 
ing each other’s efforts” (Johnson & Johnson, 1999, p. 71). When working together, they can 
improve the way of thinking and interpreting skills of one another alongside their social or per- 
sonal relations. 


Social skills: CL activities necessitate students to exhibit a number of social and interperson- 
al skills to deal with cooperative tasks. If they lack such skills, they have to be taught. Just plac- 
ing them in groups to work in cooperation with each other does not help. Being more social 
leads to more interaction and better relationships among the students. This could be improved 
by assigning them different roles during the CL activities. 


Group processing: it refers to the students’ reflections over the successes and failures they 
experience during the CL activities and identifying behaviors which need reinforcement to im- 
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prove or need to be changed by discussing within their groups to increase the efficacy of their 
contributions to accomplish a shared goal. 


To sum up, in spite of differences all of the cooperative learning methods and structures 
share one trait in common, that is, “having students work in small groups or teams to learn aca- 
demic material” (Slavin, 1991, p.71) while taking the five CL elements into consideration in the 
process as briefed above. As discussed elsewhere the CL methods provide the students enough 
opportunities to engage in extensive social interactions, which will contribute a lot in their 
learning especially in language acquisition. Although there are controversies about the contex- 
tual problem of foreign language education, which might limit the language acquisition process, 
again it is believed that when students get the right input and exposure to the target language 
will benefit them a lot. 


2.3. The present study and its significance 


The main purpose of this study was to develop a “Cooperative Learning Attitude Scale 
(CLAS) for the EFL Students” through a content validity study based on the expert review. 
However, the authors tended to examine its construct validity through confirmatory factor anal- 
ysis (CFA). The researchers did not come across any measurement tool on EFL students’ atti- 
tudes towards the use of cooperative learning in their language learning process especially with 
a focus on EFL acquisition. Attitudes are a series of beliefs and ways of judgment towards 
something, which, in turn, will shape the behaviors of an individual. It has a direct relation with 
the students’ motivation and if they hold a ‘negative attitude’, it will affect their achievement 
level. Discovering the attitudes of the students about learning strategies will allow discovering 
which instructional approaches are appropriate and helpful to them during the teaching and 
learning process (Mentz and Van Zyle, 2016). 


Besides, the scale development method used in this study might be the first of its kind in 
language education with the specifications elaborated above. Content validity studies based on 
expert review for keeping or discarding items in a measurement tool is widely used in nursing 
education (Pilot & Beck, p. 2006). The researchers thought that this would be a good approach 
to develop measurement tools for other fields of education, too; EFL education in the current 
study. 


3. Method 
3.1.Research design 


This study used a content validity approach to measurement tool development in accordance 
to the reviews of a group of content expert panel and their consensus over the representativeness 
of the variables and their significance to the content domain (Rubio, Berg-Weger, Tebb, Lee, & 
Rauch, 2003; Ayre & Scally, 2014; Zamanzadeh et al., 2015). Pilot, Beck, and Owen (2007) 
argue that “Content validity concerns the degree to which a scale has an appropriate sample of 
items to represent the construct of interest...whether the domain of content for the construct is 
adequately represented by the items” (p. 459). 


There is a variety of ways to the quantification of content validity and/or computing the 
agreement level of the content experts with regard to the relevance of items in a scale (Pilot and 
Beck, 2006; Lynn, 1986). However, the content validity index (CVI), which is extensively used 
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in nursing education, was adopted in this particular study seemingly being its first example in 
language education. As a supplementary method to calculating the consensus level of the raters, 
a modified kappa statistical test was also conducted which is elaborated later on (Pilot and 
Beck, 2006). 


3.2.Procedure 


Content validity study is carried out in two stages of development and judgment in general. 
The development stage is qualitative-based with three sub-steps, where the content domain is 
identified, the scale items are generated, and put into expert review form. In the judgment stage, 
the relevance and the clarity of the items are judged by a group of content experts to identify 
valid items. Then their CVI will be computed to quantify the judgment of the expert panel 
(Lynn, 1986). A similar procedure was followed in this study as elaborated below: 


3.3.Developing the items 


First of all, an extensive literature review was carried out on cooperative learning and lan- 
guage acquisition in order to identify the representative variables as relevant to the respective 
domains. At the initial stages of the study, 113 items were generated with a direct focus on Eng- 
lish as a foreign language (EFL) and language acquisition. Then all the items were screened for 
identifying the similar as well as unrelated items by the researchers. As a result, 32 items of 
such kinds were found and excluded from the study. After doing major revisions and rewording, 
the remaining 81 items were put into an expert review form (Lynn, 1986). This form required 
the content experts to rate the relevancy and clarity level of each item on a four-point rating 
scale from 1-4, with 4 reflecting perfect relevance of the items in the relevancy box and very 
clear in the clarity box (e.g. see: Lynn, 1985; Pilot and Beck, 2006; Pilot et al., 2007, Zamanza- 
deh et al., 2015). Besides providing the required written instructions on how to rate the items, 
the experts were requested to check the items for their accuracy and make corrections where 
necessary. In addition, they were asked to write down the possible summarizing sub-scales of 
several variables; each item considered as a distinct variable. 


3.4. Selecting the expert panel and the sample 


Nine different content experts participated in the study. Each of them had substantial skills in 
using CL in higher education and had sufficient EFL skills. Four of them had expertise in EFL 
teaching while the rest taught other fields. Eight of them had Ph.D. in curriculum development 
and instruction and one in assessment and evaluation. Of these experts, two of took participation 
in all three rounds of the expert-review. In the first round, nine experts were selected in total. 
Six expert review forms were handed over in person, while three copies were sent by email. Of 
nine forms, only five were returned which is an acceptable number for a content validity study 
based on the experts review. According to Lynn (1985), the number of content experts should 
not be less than three. However, preferring a panel of minimum five experts prevents the occur- 
rence of chance agreement problem. In the second round of expert review, again a panel of five 
content experts participated (two of them being new members) to rate the new set of revised 
items after the removal of the irrelevant ones. The third round of review required the content 
experts to assign the items to the given sub-factors suggested in the first two rounds of the ex- 
pert reviews to form the final form of the scale. As usual, five reviewers, including two new 
members, participated in the final stage of the expert review, too. 
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After the completion of expert-review procedure, the last version of the scale was responded 
by 136 pre-service EFL teachers in Turkey, including sophomores, juniors, and seniors. The 
freshmen were not included in the study since they did not have much experience in CL activi- 
ties in EFL education. The collected data from these students was used for the CFA. 


3.5. Data analysis 


The collected data in the first two rounds of the study were entered into two content validity 
index (CVI) calculation tables in each round of expert review; one for calculating the CVI for 
the relevancy of the items and another for their clarity. First, item-level CVI (I-CVI) for the 
relevancy of all the items were calculated by counting the number of the experts who give a 
rating of 3 or 4 divided by the total number of the content experts. The items having an I-CVI of 
‘1’ were retained while the ones under that floor were excluded. According to Lynn (1986), if 
the number of the content experts is five or less than that the CVI should be ‘1’, that is, every- 
one in the panel should agree for an item to be retained. However, Pilot et al. (2007) maintain, 
“an I-CVI of 0.78 or higher for three or more experts could be considered evidence of good 
content validity” (p. 459). Moreover, the scale level CVI (S-CVI) of the items were calculated 
using two approaches for both the relevancy and the clarity of the items, that is, by averaging 
the S-CVI (S-CVI/AV) by dividing the sum of I-CVI to the total number of the items and the 
universal agreement S-CVI (S-CVI/UA) was calculated by dividing the total of items with an I- 
CVI of ‘1’ to the total number of the items. A minimum 0.80 for both S-CVI/AV and S- 
CVI/UA is considered acceptable, while a value of 0.90 or over would be excellent (Pilot et al., 
2007). 


To control the chance agreement issue, which CVI does not, modified kappa tests were run 
for each item. To do this, first “the probability of chance agreement was” calculated using the 
formula presented in Pilot et al. (2007, p. 466) and given in a fitting format here in Zamanzadeh 
et al. (2015, p. 69) as “P.=[N! / A! (N - A)!]*.5‘””. N indicates the number of content experts, A 
the number of experts confirming the relevance of items to the content area by rating them as 3 
or 4. Then the modified kappa was calculated using “K = (I-CVI - Pc) / (1 - Pc)” formula (Za- 
manzadeh et al., 2015, p. 69; Pilot et al., 2007). Full kappa agreement value equals to ‘1’ while 
the chance agreement equals to zero (Viera and Garrett, 2005). In this study, the perfect modi- 
fied kappa (k =/) was preferred as an acceptable level as was in I-CVI. 


To examine whether the scale developed based on expert reviews support the construct va- 
lidity, a correlated traits model of CFA was performed via LISREL 8.71. Prior to CFA, the 
normality and linearity assumptions were ensured. The VIF (< 5) and Tolerance (> 0.20) values 
indicated the absence of singularity and multicollinearity problems. The missing values of less 
than 5% were substituted with mean scores and the negative items were reverse-coded. After- 
ward, the normed chi-square, Root Mean Square Error of Approximation (RMSEA), Normed 
Fit Index (NFID, Non-Normed Fit Index (NNFI), Comparative Fit Index (CFI), and Standardized 
Root Mean Square Residual (SRMR) fit indices. According to Pituch & Stevens (2016, p. 654), 
NFI, NNFI, and CFI of 0.90, as well as SRMR of < 0.10 and RMSEA of < 0.08, are indicative 
of adequate fit. Besides, the normed chi-square is expected to be smaller than 3 (Kline, 2016). 
By the end, the internal consistency of the measurement tool was evaluated through Cronbach’s 
alpha with a minimum acceptance level of 0.60 to 0.70 (Hair, Black, Babin, & Anderson, 2014, 
p. 90). 
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4. Results 
4.1. The first round of expert review 


In the first round of expert review, the content experts (NV = 5) provided substantial construc- 
tive written feedback on improving the comprehensibility of the items while rating the relevan- 
cy and clarity of each item. This helped the researchers in identifying and discarding several 
unrelated items. They also suggested some possible factors each representing several variables 
(i.e. items). As reported elsewhere, the collected data were put in CVI computing tables and the 
items having an I-CVI of ‘1’ were kept while the others were discarded. As a result, 29 items 
were found to be less than the acceptable measure (I-CVI<1) set for the current study and there- 
fore eliminated. So, 52 out of 81 items were retained. 


Moreover, the S-CVI/AV and S-CVI/UA were found to be 0.916 and 0.642 respectively. 
Here, the S-CVI/AV showed an excellent congruity level of the content experts over the rele- 
vancy of the items to the content domain, whereas the S-CVI/UA is inadequate. This issue was 
considered to be solved in the second round of expert review after the revision and elimination 
of the unrelated items. Further, a modified kappa test was run for each item to control for the 
chance agreement problems. The items which had a perfect I-CVI showing perfect agreement 
level of the content experts over the relevancy of individual items, showed the highest agree- 
ment level in this test, too (k=1). This indicates that the problem of relevancy chance agree- 
ment was not met in any of the items. 


The same procedure was applied in calculating the consensus of the content experts over the 
clarity or the comprehensibility of the items using CVI tests. The results showed that the majori- 
ty of the items (61) were rated as 3 or 4 showing high clarity of the items; each with an I-CVI of 
‘1’. Moreover, the S-CVI/AV and S-CVI/UA were found to be 0.946 and 0.753 showing high 
expert congruency over the clarity of the items in averaging, but not in the universal agreement 
level. 


Although some of the items, which were considered as highly relevant and comprehensible, 
some of the experts recommended some revisions. Besides, one of the items, i.e. working with 
other students, informs me of my strengths and weaknesses, was proposed to be divided into two 
separate items. Therefore, after performing the recommended revision process, a second version 
of the expert review form was devised by the inclusion of 52 items for the second round of ex- 
pert consultation. In this form, the content experts were requested to assign the items to seven 
different representative sub-scales of cooperative learning they had provided in the first round 
of their review along with the ones suggested by the researchers. Further, they were asked to 
rate the relevance and the clarity of the items once again to enable the researchers to create the 
final form of the instrument. 


4.2. The second round of expert review 


The second version of the form was structured after major revisions adding seven of the sub- 
scales suggested by five experts (i.e. Self-Efficacy in EFL, Motivation in EFL, Personal Growth 
in EFL, Language Proficiency, Language Anxiety, EFL Teacher Role, Peer Influence on EFL 
Acquisition). Besides rating the relevancy and clarity of the items, the content experts assigned 
each item to the given sub-scales. The CVI and modified kappa calculation were run for a sec- 
ond time for both the relevancy and clarity of the items. The results show that all the items, but 
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one, meet the excellent congruency of the reviewers considering the relevancy and clarity of 
each item (I-CVI=1, k=l). As shown in Table 1, the S-CVI/AV and S-CVI/UA calculations 
also showed an excellent level of consensus (0.996 and 0.981 respectively). The only item, 
which did not meet the perfect agreement of the experts (I-CVI=0.8, k'=0.76) was discarded. 
Moreover, two items related to the EFL Teacher Role were discarded, as they were less than 
three to be placed under a separate sub-scale. Therefore, this sub-scale was excluded. As a re- 
sult, 49 items were included in the final scale form. 


However, while assigning the items to the given sub-scales, the expert panel assigned each 
item to more than one sub-scale. Some of them suggested merging some of the subscales to 
prevent this confusion. Therefore, the number of the sub-scales was decreased to five; some of 
them being named differently along with their definitions, to go for the third round of expert 
review to only the assign the items to the new sub-scales (Self-Efficacy in EFL, Language Pro- 
ficiency, Self-Regulation in EFL, Motivation in EFL, and Affective Attitudes). 


4.3. The third round of expert review 


In this phase, five reviewers assigned the 49 items to five different subscales noted above. 
Items assigned to a subscale by at least three reviewers were retained accordingly. From these 
items, four of them were assigned to different subscales, where the reviewer consensus was not 
met and therefore were eliminated. Table 1 contains the final list of 45 items that met three to 
five matches of item-factor assignment with the resultant content validity statistics obtained 
from the second expert review. 


Table 1. The final content validity test results of CLAS items based on the ratings of five 
content experts after they assigned the items to the relevant sub-scales 


# of Ex- 
Items perts rated Tyre pe##  K#H 
items as 3 
or 4 
Self-Efficacy in EFL 
1. Working together on a language-related task leads to the 5 1 0.03125 1 
self-improvement in EFL skills. 
2. Teamwork with the EFL students increases my self- 5 1 0.03125 1 
confidence in language use. 
3. lam afraid of making errors when working with the EFL 5 1 0.03125 1 
students. 
4. Working in small groups enables me to take a risk in ex- 5 1 0.03125 1 
pressing myself in the target language. 
5. I learn things better in peer interaction with the EFL stu- 5 1 0.03125 1 
dents other than on my own. 
6. Helping my friends learn contributes to my personal 5 1 0.03125 1 


growth in EFL. 


Language Proficiency 
7. Exchanging ideas with EFL students improves my com- 5 1 0.03125 1 
municative language skills. 


6. Cooperating in learning leads to a better understanding of 5 1 0.03125 1 
the language lessons. 
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# of Ex- 
Items pertsrated Toypt  pet# = K#H 
items as 3 
or 4 
9. Peer interaction boosts up my overall English language 5 1 0.03125 1 
achievement. 
10. I learn new language patterns when I work together with 5 1 0.03125 1 
my peers. 
11. Learning about the experiences of my classmates makes 5 1 0.03125 1 
me strengthen my own language skills. 
12. Interaction with EFL students contributes to the enrich- 5 1 0.03125 1 
ment of my vocabulary. 
13. I believe my overall reading skills improve when I work 5 1 0.03125 1 
with my peers. 
14. My listening skill improves in small group discussions. 5 1 0.03125 1 
15. My speaking skill improves in small group discussions. 5 1 0.03125 1 
16. [learn from my group members when I work with them 5 1 0.03125 1 
on a common language-related task. 
17. Face-to-face communication in small groups gives real 5 1 0.03125 1 
language practice. 
18. I can acquire English better in an interactive learning 5 1 0.03125 1 
environment. 
19. Working with my classmates allows me to improve my 5 1 0.03125 1 
writing skill. 
20. Language acquisition process becomes meaningful 5 1 0.03125 1 


through shared group tasks. 
Self-Regulation in EFL 


21. Peer interaction informs me of my strengths in the lan- 5 1 0.03125 1 
guage acquisition process. 


22. Peer interaction informs me of my weaknesses in the 5 1 0.03125 1 
language acquisition process. 


23. Participating in small group works helps me generate new 5 1 0.03125 1 
ideas leading to meaningful language practice. 

24. Sharing pieces of work gives me a sense of responsibility 5 1 0.03125 1 
towards my own language acquisition process. 

25. Interaction with EFL students contributes to my cognitive 5 1 0.03125 1 
development. 

26. Sharing responsibilities in small groups promote my deci- 5 1 0.03125 1 


sion-making skills in how to complete a given task in the 
target language. 


27. Working with EFL students helps me evaluate my own 5 1 0.03125 1 
learning process. 

Motivation in EFL 

28. I get motivated to feel responsible for promoting my lan- 5 1 0.03125 1 
guage skills during the cooperative group works. 


29. Cooperation in group works makes me believe in my 5 1 0.03125 1 
abilities in language acquisition. 
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# of Ex- 
Items perts rated Toye pet# KH 
items as 3 
or 4 
30. I feel motivated when I contribute to the accomplishment 5 1 0.03125 1 
of a shared group task aiming to improve my own English. 
31. Working with higher-level EFL students motivates me in 5 1 0.03125 1 


learning. 
Affective Attitude 


32. Cooperating with my classmates in fulfilling given lan- 5 1 0.03125 1 
guage practice tasks gives me a sense of community. 


33. Language classes become interesting by working together. 5 1 0.03125 1 
34. Cooperating with the EFL students gives me a sense of 5 1 0.03125 1 
significant self during the language acquisition process. 

35. Interaction with EFL students negatively affects my own 5 1 0.03125 1 
academic performance. 

36. I feel relaxed in practicing English with my peers. 5 1 0.03125 1 
37. Working with EFL students helps me socialize, which 5 1 0.03125 1 
will provide more opportunities to practice my English in 

return. 

38. My language anxiety level increases when working on a 5 1 0.03125 1 
group task. 

39. Active participation in a small group discussion eases my 5 1 0.03125 1 
language acquisition process. 

40. I feel frustrated when an EFL student in my group cannot 5 1 0.03125 1 
fulfill the task of his/her part. 

41. Language classes become too noisy when we engage in 5 1 0.03125 1 
small group activities. 

42. I like to share what I know with the other EFL students. 5 1 0.03125 1 
43. I learn about how to work with different people when 5 1 0.03125 1 


cooperating with my EFL classmates. 


44. Working together encourages individual tolerance 5 1 0.03125 1 
amongst the EFL students. 


45. I learn new language features quickly when my class- 5 1 0.03125 1 
mates explain them. 


S-CVI/Ave= 0.996, S-CVI/UA=0.981 
Note: the items in bold were later eliminated as a result of CFA. 
*Item level content validity index, **Probability of chance agreement, ***Modified kappa 


4.4. Confirmatory factor analysis 


The preliminary CFA results indicated that almost all of the fit statistics were of adequate 
cutoff, x° (N = 136) = 1604.34, df= 934, p < 0.001; x°/df= 1.71, NFI = 0.86, NNFI (TLI) = 0.93, 
CFI = 0.93, SRMR = 0.079, RMSEA = 0.073. Only NFI was under the threshold. Moreover, the 
t statistics belonging to four items were insignificant and two others had a factor loading of 
smaller than 0.30. They were later removed as two content experts suggested to. These prob- 
lems are probably because of the small sample size in the study since CFA requires a larger 
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sample, say, 200 or more. Aside from these three issues, the fit indices suggested an acceptable 
measurement model without modification. 


Table 2. Standardized estimates, standard errors, ¢t values, item-total correlations, and 
Cronbach’s Alpha 


: Item-total corre- 
Factors Items Lie SE t ao. ae a. 
1 051 0.74 5.81" 0.514* 
_ 2 0.58 0.66 6.80" 0.553" 
ees 4 0.33 0.89 3.66% 0.314* 0.68 
5 0.61 0.62 7.25% 0.561* 
6 0.60 0.64 7.02* 0.551* 
7 0.64 0.58 8.18% 0.592* 
8 0.65 0.58 8.26% 0.622* 
9 0.62 0.61 7.85% 0.617% 
10 0.57 0.68 6.95% 0.555% 
i 0.60 0.65 7.40* 0.573" 
12 0.54 0.71 6.59% 0.503" 
aoe RIO ag 0.49 0.76 5.89* 0.525* 0.86 
15 0.51 0.74 6.21* 0.525% 
16 0.67 0.55 8.58% 0.650* 
17 0.55 0.70 6.72" 0.534* 
18 0.47 0.78 5.59% 0.520% 
19 0.39 0.85 4.51% 0.432* 
20 0.65 0.58 8.23* 0.668" 
21 0.74 0.45 9.778 0.677 
22 0.43 0.81 5.08" 0.391% 
ch “88 0.71 0.50 9.16* 0.682" 
oe ae aa in 44 0.65 0.58 8.15* 0.640* 0.82 
25 0.68 0.54 8.66% 0.614* 
26 0.63 0.60 7.96 0.594* 
27 0.60 0.64 7.AT* 0.639% 
28 0.77 0.41 10.04* 0.676 
ES ES 
ee ee 
31 0.45 0.80 5.21* 0.444* 
32 0.62 0.61 7.78% 0.586" 
33 0.61 0.63 7.61* 0.629% 
34 0.68 0.54 8.73* 0.642* 
36 0.62 0.62 1.73% 0.619* 
ES ok 
Affective Attitude . one oe eee 0.87 
42 0.63 0.60 7.99% 0.616% 
43 0.65 0.58 8.20% 0.613* 
44 0.56 0.69 6.80" 0.563" 
45 0.55 0.70 6.71% 0.532* 
*p<0.01 


Furthermore, the statistics were examined by discarding four items (item 3, 38, 40, and 41) 
with insignificant ¢ test values and two others (item 14 and 35) with low factor loadings of < 
0.30. Four items, 1 and 2 as well as 13 and 19, were modified by sharing their error variances. 
The resultant fit indices displayed a considerable improvement within acceptable criteria and all 
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items had significant ¢ values (x° (NV = 136) = 1252.75, df= 690, p < 0.001; x°/df= 1.81, NFI = 
0.90, NNFI = 0.95, CFI = 0.95, SRMR = 0.072, RMSEA = 0.071). As seen in Table 2, the fac- 
tor loadings of the observed variables ranged from 0.33 to 0.77 and the item-total correlations 
were significant across all variables. 


The internal consistency reliability of the subscales of the CLAS tested through Cronbach’s 
alpha varies between 0.68 and 0.87. However, the alpha for the entire scale was excellent (a = 
0.95). To conclude, the coefficient alphas of 0.60 to 0.70 show minimum level of adequacy 
(Hair et al., 2014, p. 90). 


5. Discussion and conclusion 


Measurement tool development as a complex process requires substantial time and resources 
to collect data several times. Unfortunately, this may not be always feasible for the researchers. 
Thus, the emergence of alternative methods, such as the one used in the first phase of this study, 
could minimize the hustle and bustle of research tool development. Herewith, instead of em- 
ploying an exploratory factor analysis, the CLAS was developed through a three-round rigorous 
expert review. All the analyses related to content validity were carried out according to the 
prominent researchers in this respect (e.g. Lynn, 1986; Rubio et al., 2003; Pilot and Beck, 2006; 
Pilot et al., 2007). 


In the first round of expert review, the numbers of items were reduced to 52 out of 81 based 
on the CVI and modified kappa statistics computed for individual items. The remaining items 
were subjected to major revision based on the written comments provided by the expert panel 
and the low S-CVI/UA values for both the relevance and the clarity of the items. The second 
expert review form, containing 52 revised items, was devised along with the seven subscales 
suggested by the reviewers in the first round. 


The second-round expert review provided positive results with regard to the relevance of the 
items to the content domain and their comprehensibility at an excellent level. In this process, 
three items were eliminated; one because of low CVI and two others were not enough to form a 
sub-scale. As a result, 49 items were retained. Further, the confusion arising from the similarity 
of the suggested factor names caused some of the experts to assign one item to more than one 
factor. Therefore, the factors were revised and reduced to five as recommended. So doing, the 
third round of expert review was conducted with 45 items to be assigned to five different factors 
(i.e. Self-Efficacy in EFL, Language Proficiency, Self-Regulation in EFL, Motivation in EFL, 
and Affective Attitude). This time the reviewers only assigned the items to the relevant sub- 
scales without rating their relevance or clarity. Of 49 items, 45 that were assigned to the sub- 
scales by at least three experts were retained, while four of them were eliminated since they did 
not meet reviewer consensus. 


Furthermore, the construct validity of the remaining 45 items, representing five factors, was 
examined through CFA based on data from a small sample of 136 EFL pre-service teachers. 
The first run of CFA suggested the elimination of four items with insignificant t values and two 
with low standardized estimates. Of these eliminated items (i.e. 3, 14, 35, 38, 40, & 41), five of 
them exclusive of item 14, were reverse-coded before CFA since they were negative. Despite 
this, the fit statistics were within acceptable ranges except one. However, the second run of 
CFA, by excluding the stated items, suggested a good-fitting model with adequate factor load- 
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ings. The ¢ values and item-total correlation were significant as well as the coefficient alpha 
indicated good internal consistency. The final form of the instrument had 39 items in total with 
five sub-scales. Self-Efficacy in EFL represented five, Language Proficiency 13, Self-Regulation 
in EFL 7, Motivation in EFL 4, and Affective Attitude 10 items. 


This measurement instrument has utility only in measuring university EFL students’ attitudes 
towards CL based on five response categories of strongly agree, agree, undecided, disagree, and 
strongly disagree. According to the last form of the instrument, the total score that can be ob- 
tained from the scale may vary between 39 and 195. Similarly, the total factor scores may range 
from five to 25 for the first factor, 13 to 65 for the second, seven to 35 for the third, four to 20 
for the fourth, and 10 to 50 for the fifth. However, the interpretations are recommended to be 
made based on the mean scores. 


To sum up, the CLAS developed based on the expert review (by conducting different CVI 
tests) and examined through CFA, yielded adequate evidence supporting both the content validi- 
ty and the construct validity. Some of the problems that caused item elimination according to 
CFA results could be attributed to the small sample size. Therefore, further research with a larg- 
er sample may require the inclusion of the eliminated items and recheck fit indices. Table 1 
contains all the 45 items including the ones eliminated, while Table 2 provides the retained 39 
items after item elimination was carried out based on CFA results. 
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